1. Field of the Invention
The present invention relates to an image processing device, an image processing method, and a recording medium, and in particular, to an image processing method in which the gray balance of an original image recorded on a photographic film is estimated, an image processing device to which this image processing method is applied, and to a recording medium on which is recorded a program for operating a computer as the image processing device.
2. Description of the Related Art
The color balance of an image which is made visible on a photographic film by carrying out processings such as development for a photographic film with an object photographed and recorded thereon by a camera or the like (hereinafter simply referred to as an image recorded on the photographic film) is influenced by characteristics of the photographic film itself, processing conditions for developing processing, and the like (hereinafter generically referred to as film characteristics) and is deviated or biased from the color balance of the object at the time of photographing. Therefore, when the image recorded on the photographic film is recorded onto a recording material such as a photographic printing paper or is displayed on display means such as a display, the color balance of the image to be recorded onto the recording material or the image to be displayed on the display means (hereinafter generically referred to as the output images) must be corrected such that the color balance of the output image corresponds to the color balance of the object at the time of photographing (such that gray portions of the object at the time of photographing are reproduced as gray in the output image).
As an example of methods of correcting the color balance of output images, a method is known in which pixels corresponding to highlight regions of an image (i.e., an original image) recorded on a photographic film (e.g., pixels having a maximum density in a negative image, and pixels having a minimum density in a positive image) are assumed to be white, while pixels corresponding to shadow regions of the original image (e.g., pixels having a minimum density in the negative image, and pixels having a maximum density in the positive image) are assumed to be black. The gray balance representing the color balance, in the original image, of gray portions of the object at the time of photographing is estimated, and the color balance of an output image is corrected on the basis of the estimated gray balance. In this method, pixels corresponding to highlight regions and pixels corresponding to shadow regions are respectively extracted from the original image. Then, for example, an axis which connects, by a straight line, the points on an RGB density coordinate system which correspond to both of the pixels is determined as the gray axis representing the gray balance.
However, in the aforementioned correction method, when the color of the pixels corresponding to the highlight regions in the original image is not white, as can be seen, for example, in an image in which a person has been photographed by using an electronic flash and cheek portions of the person""s face (i.e., flesh-colored portions) are highlighted, the gray axis representing the appropriate gray balance cannot be determined (the same holds for the shadow regions). Further, the color balance of the output image deviates to a complementary color for the color of the pixels corresponding to the highlight regions. This deviation is referred to as xe2x80x9chighlight failurexe2x80x9d. The frequency of generation of an original image susceptible to the above-mentioned highlight failure is relatively high, and a drawback arises in that the probability of obtaining an output image with an appropriate color balance is low.
Further, a method is also known in which, on the basis of Evans"" theory, an image having a constant LATD (i.e., light accumulated transmission density) for each of component colors (e.g., R, G, and B) is regarded as an image whose color balance corresponds to the color balance of an object at the time of photographing (i.e., an image having an appropriate gray balance), and the LATD for each of the component colors (e.g., R, G, and B) of the original image is measured. The color balance of an output image is corrected so that the LATD for each of the component colors of the output image becomes constant.
However, in the aforementioned correction method, in a case in which a non-gray region having a substantially constant hue (e.g., a region corresponding to green turf, blue sky, blue sea, or the like) occupies a relatively wide area of the original image, when correction is carried out so that the LATD for each of the component colors becomes constant, the color balance of the output image deviates to the complementary color of the aforementioned region (this deviation is referred to as xe2x80x9ccolor failurexe2x80x9d). The frequency of generation of an original image susceptible to the above-described color failure is relatively high. Therefore, a drawback arises in that, as in the case with the other correction method described above, the probability of obtaining an output image with an appropriate color balance is low.
Moreover, Japanese Patent Application Laid-Open (JP-A) No. 9-83825 discloses a linear transformation technique. In this technique, image data DRxe2x80x2, DGxe2x80x2, and DBxe2x80x2 regarding low saturation pixels is acquired from image data DR, DG, and DB representing an original image. Shadow points DRs, DGs, and DBs and highlight points DRh, DGh, and DBh in the image data DRxe2x80x2, DGxe2x80x2, and DBxe2x80x2 are respectively determined. Regarding one of a pair of the image data (DRxe2x80x2, DGxe2x80x2) corresponding to each other for each pixel, values in the other are averaged for each of the same values in the one to obtain a set of image data (DRxe2x80x3, DGxe2x80x3). Subsequently, a relationship between the densities of the two colors (or R, G) is determined from the set of the image data (DRxe2x80x3, DGxe2x80x3), the shadow points (DRs, DGs), and the highlight points (DRh, DGh). On the basis of this relationship, linear transformation is carried out so that at least one of the image data DR and DG is made equal to the other over the entire region thereof.
By excluding data of high saturation pixels from the image data representing the original image from the object of computation, the above-described technique is meant to reduce effects caused by the original image susceptible to color failure. In addition to the high saturation pixels, pixels which are adjacent to the high saturation pixels and whose hue difference with respect to the high saturation pixels is within a predetermined range of values, are also excluded from the object of computation. However, in practice, even if the above-mentioned processes are effected with respect to the original image susceptible to color failure, among the pixels of non-gray regions having a substantially constant hue which occupy a relatively wide area of the original image, many pixels remain without being excluded from the object of computation. Therefore, a drawback arises in that correction accuracy of the color balance for the output image is not sufficient when the original image is susceptible to color failure.
Further, JP-A No. 5-297473 discloses a technique in which a large volume of image information obtained by reading images recorded on various types of photographic films is accumulated and stored for each type of the photographic film, and when an image recorded on a particular type of photographic film is to be recorded by exposing a photographic printing paper, exposure conditions are determined by regarding, as a representation of the gray balance, the color balance corresponding to the average of the image information accumulated and stored with regard to the particular film type.
In the above technique, however, the accuracy of the color balance corresponding to the average of the accumulated and stored image information (i.e., deviation with respect to the gray balance) depends on the contents of the image information accumulated and stored in each exposure device. Therefore, in a case in which the same image is recorded by exposing photographic printing papers using different exposure devices, the color balances of the prints obtained by recording the image by exposing the photographic printing papers are different in the respective exposure devices. Further, even in a case in which an image is recorded by exposing photographic printing papers with the same exposure device, when the same image is recorded by exposing photographic printing papers with a time interval at a plurality of times, such as in printing with image processing and extra printing, there arises a drawback in that the color balances of prints obtained differently with respect to such a time differ from each other because of effects of image information newly stored and recorded during the interval.
In view of the aforementioned, an object of the present invention is to obtain an image processing device, an image processing method, and a recording medium, in which the gray balance can be accurately and stably estimated from image data of an original image recorded on a photographic film.
In order to achieve the above-mentioned object, a first aspect of the present invention is an image processing device, comprising: extracting means which, on the basis of image data expressing a single original image recorded on a photographic film, determines an image characteristic value of the original image and, on the basis of the determined image characteristic value, extracts from the image data gray candidate pixels which have a high possibility of being pixels corresponding to gray portions of an object, the determination and extraction being carried out for each of a plurality of original images recorded on the photographic film; and estimating means for estimating gray balance on the basis of distribution of a group of gray candidate pixels on a predetermined coordinate system which group comprises the gray candidate pixels respectively extracted from the plurality of original images by the extracting means.
The extracting means according to the first aspect, on the basis of image data expressing a single original image recorded on a photographic film, determines an image characteristic value of the original image and, on the basis of the determined image characteristic value, extracts from the image data gray candidate pixels which have a high possibility of being pixels corresponding to gray portions of an object, the determination and extraction being carried out for each of a plurality of original images recorded on the photographic film. In this way, even if the original image is an image wherein a relatively wide area is occupied by, for example, a non-gray region having a substantially constant hue (i.e., an image susceptible to color failure), the majority of the pixels in the aforementioned region is excluded from the gray candidate pixels, and pixels which have a high possibility of being pixels corresponding to gray portions of the object, i.e., pixels which have a high possibility of expressing the gray balance, are extracted as the gray candidate pixels.
The gray candidate pixels can be extracted in such a manner that, for example, saturation of each of the pixels is used as an image characteristic value of the original image, the saturation of each of the pixels is compared with a predetermined value and high saturation pixels are determined, and pixels remaining after the high saturation pixels have been excluded are regarded as the gray candidate pixels.
Moreover, the following case is also possible. Namely, the maximum density and the minimum density in the original image are respectively extracted for each of the component colors from image data. On a predetermined coordinate system (e.g., a three-dimensional orthogonal coordinate system whose coordinate axes are density values for the respective component colors), a first reference axis is set which connects a density point corresponding to the maximum density and a density point corresponding to the minimum density for each of the component colors. Subsequently, on the basis of the distance on the predetermined coordinate system between a point corresponding to each of the pixels and the first reference axis, pixels whose distance to the first reference axis is relatively large are determined to be high saturation pixels. In this case, the maximum density and the minimum density in the original image are the image characteristic values.
Further, after the high saturation pixels have been extracted in the aforementioned manner, determination is made as to whether hue (or hue and saturation) of pixels existing in the neighborhood of the high saturation pixels on the image approximate that of the high saturation pixels. It is preferable that the pixels determined to be similar in hue to the high saturation pixels are also excluded, as pixels that are not gray candidate pixels. In this way, even if the original image is an image susceptible to color failure, most of the pixels in the aforementioned region are excluded from the gray candidate pixels. In this case, the high saturation pixels and the positions and hue of the respective pixels are the image characteristic values.
Furthermore, the estimating means according to the first aspect estimates the gray balance on the basis of the distribution on the predetermined coordinate system of a group of the gray candidate pixels comprising the gray candidate pixels respectively extracted from a plurality of original images by the extracting means. If an original image whose image contents are liable to cause reduction in estimated accuracy of the gray balance (e.g., an image susceptible to highlight failure or color failure, and the like) is included among the plurality of original images, the gray candidate pixels extracted from the original images may often be pixels which do not express the gray balance.
However, the group of the gray candidate pixels comprises the gray candidate pixels which have been respectively extracted from the plurality of original images recorded on the same photographic film. Therefore, variation in image contents of the plurality of original images are averaged for the entire group of the gray candidate pixels. Even if pixels which are appropriate as the gray candidate pixels (pixels expressing the gray balance) have not been extracted from some of the plurality of original images, the distribution of the group of the gray candidate pixels on the predetermined coordinate system expresses with a high degree of accuracy, on the whole, the gray balance of each of the original images recorded on the photographic film (the gray balance which reflects the film characteristics and the like). This is also confirmed by experiments conducted by the present inventor.
The estimating means estimates the gray balance on the basis of the group of the gray candidate pixels on the predetermined coordinate system. Accordingly, the gray balance can be estimated with a high degree of accuracy from the image data of the original images recorded on the photographic film. Further, in accordance with the first aspect, there is no need to accumulate and store the data of original images which have been processed in the past and use the data at the time of estimating the gray balance. Therefore, mass storage means for storing data is not necessary. The results of estimation of the gray balance from (the plurality of original images recorded on) the same photographic film are made stable (substantially stable all the time) without being affected by the contents of accumulated and stored data.
Specifically, estimation of the gray balance by the estimating means from the distribution of the group of the gray candidate pixels on the predetermined coordinate system can be realized by, for example, determining a characteristic line (which may be a straight line or a curve) from information on the positions of the respective gray candidate pixels, which form the group of the gray candidate pixels, on the predetermined coordinate system by using a technique such as the method of least squares or the like.
Further, at the time a characteristic line expressing the gray balance is determined as in the aforementioned, it is preferable that the extracting means is structured so as to carry out the following processings on each of the original images: after the gray candidate pixels have been extracted, the maximum density and the minimum density are respectively extracted from the gray candidate pixels for each of the component colors; a second reference axis is set which connects a density point corresponding to the maximum density for each of the component colors and a density point corresponding to the minimum density for each of the component colors on the predetermined coordinate system; and, on the basis of the distances on the predetermined coordinate system between points respectively corresponding to the gray candidate pixels and the second reference axis, a weight is applied to the respective gray candidate pixels such that the weight increases as the distance decreases. The characteristic line is preferably determined by taking into consideration the weight applied to the respective gray candidate pixels. In this way, estimated accuracy of the gray balance can further be improved.
The distribution of the group of the gray candidate pixels on the predetermined coordinate system expresses the gray balance, which reflects the film characteristics and the like. In general, however, photographic films have a characteristic in that color formation density changes in a non-linear manner in accordance with the change in the exposure amount, and therefore, the characteristic line expressing the gray balance is expressed as a curve on the predetermined coordinate system. Thus, even at the time of estimating the gray balance by the estimating means, strictly speaking, it is preferable that a characteristic curve expressing the gray balance is determined by subjecting the distribution of the group of the gray candidate pixels on the predetermined coordinate system to non-linear approximation of higher order. However, processings are considerably complicated, and deterioration in estimated accuracy may instead be caused by carrying out non-linear approximation of higher order.
In general, a color formation density area of a photographic film has a density value range of approximately 3.0, while a density area of each original image actually recorded on the photographic film has a density value range of about 1.0. Since the density area of each original image recorded on a single photographic film is unstable, there are often cases in which the group of the gray candidate pixels is distributed on the predetermined coordinate system with a scope corresponding to the color formation density area and this distribution expresses the gray balance over the entirety of the color formation density area of the photographic film. However, information on the gray balance required for each original image at the time of correcting the gray balance is only a part of the entire color formation density area of the photographic film (i.e., the density area of the each original image).
The present inventor, in view of the aforementioned, have achieved a second aspect in which, when a density area of each original image is utilized, the gray balance can be estimated with a high degree of accuracy and in a stable manner, by determining a characteristic line (gray axis) expressing the gray balance by carrying out linear approximation for the distribution of the group of the gray candidate pixels on the predetermined coordinate system.
In the second aspect, for each of the plurality of original images, by subjecting, among the distribution of the group of gray candidate pixels on the predetermined coordinate system, a distribution in a density area of an original image, which is an object of estimation, to linear approximation, the estimating means estimates a gray axis expressing the gray balance in the original image. Therefore, for each of the original images, the gray balance (gray axis) in the density area (the density area of the respective images) required for gray balance correction can be estimated with a high degree of accuracy by simple processings.
A third aspect according to the second aspect is characterized in that the estimating means carries out the linear approximation in such a manner that gradient of the gray axis is within a predetermined range.
Since the gradient of the characteristic line expressing the gray balance on a predetermined coordinate system is determined by the film characteristics and the like, the gradient falls into a certain range in almost all cases. In the present invention according to the third aspect, linear approximation is carried out such that the gradient of the gray axis is within a predetermined range, and therefore, when carrying out the linear approximation, deviation of the gradient of the gray axis from the predetermined range is restricted. Accordingly, for example, even when the gradient of the gray axis, which is determined by linear approximation from the distribution of the group of the gray candidate pixels on the predetermined coordinate system, is extremely increased or decreased for some reason, the estimated accuracy of the gray balance can be prevented from becoming significantly reduced.
A fourth aspect according to the third aspect is characterized in that the estimating means changes the restriction on the gradient of the gray axis in accordance with the number of the original images recorded on the photographic film.
Accuracy of the gray balance which is estimated on the basis of the distribution of the group of the gray candidate pixels on the predetermined coordinate system depends on the number of original images from which the gray candidate pixels have been extracted, i.e., the number of original images recorded on the photographic film. As the number of the original images decreases, the estimated accuracy of the gray balance decreases. In the fourth aspect, the restriction on the gradient of the gray axis is changed in accordance with the number of the original images recorded on the photographic film. Therefore, for example, as the number of the original images recorded on the photographic film decreases (i.e., as the estimated accuracy of the gray balance decreases), the restriction on the gradient of the gray axis can be made stricter (i.e., tolerance of the gradient can be made narrow). Regardless of the number of the original images recorded on the photographic film, the gray balance can be estimated with an accuracy of a predetermined value or more.
A fifth aspect according to the third aspect is characterized in that the estimating means changes the restriction on the gradient of the gray axis in accordance with the number of the gray candidate pixels extracted from the image data.
As described above, the distribution of the group of the gray candidate pixels on the predetermined coordinate system represents the gray balance which reflects the film characteristics and the like. The accuracy of the gray balance represented by the distribution of the group of the gray candidate pixels is correlated with the number of the gray candidate pixels which form the group of the gray candidate pixels. As the number of the gray candidate pixels which have been extracted from image data decreases, the accuracy of the gray balance represented by the distribution of group of the gray candidate pixels decreases. In accordance with this decrease, the estimated accuracy of the gray balance on the basis of the distribution of the group of the gray candidate pixels also decreases.
With regard to the above, in the present invention according to the fifth aspect, the restriction on the gradient of the gray axis is changed in accordance with the number of the gray candidate pixels which have been extracted from image data. Therefore, for example, as the number of the gray candidate pixels which have been extracted from image data decreases (i.e., as the estimated accuracy of the gray balance decreases), the restriction on the gradient of the gray axis can be made stricter (i.e., tolerance of the gradient can be made narrow). Thus, the gray balance can be estimated with an accuracy of a predetermined value or more regardless of the number of the gray candidate pixels which have been extracted from image data.
A sixth aspect according to the third aspect is characterized by further comprising designating means for designating the restriction on the gradient of the gray axis, wherein the estimating means changes the restriction on the gradient of the gray axis in accordance with a designation by the designating means.
A so-called surface exposure method is a method in which an image is recorded on a recording material by irradiating light on a photographic film such that the light transmitted through the photographic film exposes a recording material such as a photographic printing paper. When an image is recorded using the surface exposure method, density correction with respect to a recorded image which corresponds to parallel displacement of the exposure amount-color forming density characteristics along the density axis can be realized by changing the amount of insertion of a light adjustment filter onto an optical path. However, it is difficult to carry out density correction for the recorded image which corresponds to a change in the gradient of the exposure amount-color forming density characteristics.
Thus, on the basis of the results of estimation of the gray balance, in an aspect in which image data representing an original image is converted into image data having a well-balanced gray balance, and an image is outputted by using the converted image data (i.e., an image is recorded onto a recording material, an image is displayed on display means, image data are stored in an information storage medium, or the like), for example, in a case in which an output image having image quality similar to that obtained in the surface exposure method is desired, it is preferable that, when estimating the gray balance, the variation range of the gradient of the gray axis is small.
Further, as for a photographic film in which original images recorded thereon have undergone time-induced deterioration such as color fading, the film characteristics thereof are considerably different from original film characteristics. Therefore, the gray balance in the original images are considerably different from original gray balance. It is also possible that, if the restriction on the gradient of the gray axis is made strict, the estimated accuracy of the gray balance conversely deteriorates.
With respect to the above, in the present invention according to the sixth aspect, there is provided a designating means for designating the restriction on the gradient of the gray axis, and the restriction on the gradient of the gray axis is changed in accordance with a designation by the designating means. Therefore, for example, when an output image having image quality similar to that obtained in the surface exposure method is desired, by an operator designating, via the designating means, a restriction such that the restriction on the gradient of the gray axis is made strict (i.e., tolerance of the gradient is made narrow), the variation range in the gradient of the gray axis can be made small such that an output image having desired image quality can be obtained. Further, in a case in which the photographic film has undergone time-induced deterioration, by the operator designating, via the designating means, a restriction such that the restriction on the gradient of the gray axis is made lenient (i.e., tolerance of the gradient is made wide), reduction of the estimated accuracy of the gray balance, which is caused by the gray balance of the original image being different from the original gray balance, can be prevented. Accordingly, in accordance with the present invention according to the sixth aspect, estimation of the gray balance can be freely controlled.
A seventh aspect according to the third aspect is characterized in that the estimating means changes the restriction on the gradient of the gray axis in accordance with type of the photographic film.
In general, negative films are designed in such a manner that the exposure amount-color forming density characteristics in a normal exposure region is linear and the gradient thereof is substantially constant mutually among the component colors (i.e., a plurality of characteristic lines are substantially parallel to one another, each characteristic line representing the exposure amount-color forming density characteristics for one of the component colors). However, depending on the film type of negative films, the gradient of the exposure amount-color forming density characteristics may be different for each of the component colors. The gray balance of an original image recorded on such a photographic film is greatly different from the gray balance of an original image recorded on a normal photographic film. Therefore, if the restriction on the gradient of the gray axis is made strict, the estimated accuracy of the gray balance may deteriorate.
With respect to the above, the present invention according to the seventh aspect changes the restriction on the gradient of the gray axis in accordance with the film type of the photographic film. Therefore, for example, in a case where the photographic film on which original images are recorded is the type of film in which the gray balance of original images is greatly different from that of other film types, the restriction on the gradient of the gray axis can be made lenient. As a result, there can be prevented deterioration of the estimated accuracy of the gray balance of the original image recorded on the photographic film of a particular film type.
An eighth aspect according to the third aspect is characterized in that the estimating means changes the restriction on the gradient of the gray axis in accordance with exposure level at which the original image has been recorded by exposing the photographic film.
In general, the exposure amount-color forming density characteristics of a photographic film are linear in a normal exposure region and are non-linear in an underexposure region and an overexposure region. Accordingly, in estimation of the gray axis in the normal exposure region, if the variation range of the gradient of the gray axis is made large (i.e., if the restriction on the gradient of the gray axis is made lenient), the estimated accuracy of the gray balance may deteriorate. Conversely, in estimation of the gray axis in the underexposure region or the overexposure region, if the restriction on the gradient of the gray axis is made strict, the estimated accuracy of the gray balance deteriorates.
With respect to the above, in the present invention according to the eighth aspect, the restriction on the gradient of the gray axis is changed in accordance with the exposure level at which the original image has been recorded by exposing the photographic film. Accordingly, for original images having different exposure regions, the gray balance (the gray axis) in each of the original images can be estimated with a high degree of accuracy.
For example, in a case in which information expressing the exposure level is recorded onto a predetermined information recording section when original images are recorded by exposing the photographic film, the exposure level at which the original images have been recorded by exposing the photographic film can be detected by reading, from the information recording section, information which expresses the exposure level. In addition, the exposure level can be estimated, for example, on the basis of the difference in density between the film-based density and the density of the original images.
In accordance with a ninth aspect of the present invention, there is provided an image processing method comprising the steps of: (a) on the basis of image data expressing a single original image recorded on a photographic film, determining an image characteristic value of the original image and, on the basis of the determined image characteristic value, extracting from the image data gray candidate pixels which have a high possibility of being pixels corresponding to gray portions of an object, the determination and extraction being carried out for each of a plurality of original images recorded on the photographic film; and (b) estimating gray balance on the basis of distribution on a predetermined coordinate system of a group of gray candidate pixels which group comprises the gray candidate pixels respectively extracted from the plurality of original images. Therefore, the gray balance can be estimated in a highly accurate and stable manner from the image data of the original images recorded on the photographic film, as is the case with the first aspect of the present invention.
In accordance with a tenth aspect of the present invention, there is provided a recording medium on which is recorded a program for a computer to execute processings comprising: a first step in which, on the basis of image data expressing a single original image recorded on a photographic film, an image characteristic value of the original image is determined and, on the basis of the determined image characteristic value, gray candidate pixels which have a high possibility of being pixels corresponding to gray portions of an object are extracted from the image data, the determination and extraction being carried out for each of a plurality of original images recorded on the photographic film; and a second step in which gray balance is estimated on the basis of distribution of a group of gray candidate pixels on a predetermined coordinate system which group comprises the gray candidate pixels respectively extracted from the plurality of original images.
On the recording medium according to the tenth aspect, there are recorded the processings including the aforementioned first and second steps, i.e., a program for operating the computer as the image processing device according to the first aspect. Therefore, by the computer reading and executing the program recorded on the aforementioned recording medium, the gray balance can be estimated in a highly accurate and stable manner from the image data of the original images recorded on the photographic film, as is the case with the present invention according to the first aspect (and the ninth aspect).